CellFarm: An Overlay for Supporting Replication and Load Distribution in Large-Scale Systems

In this paper we propose a novel overlay network, named CellFarm, which features an unique topology composed of cliques of nodes, highly connected among themselves. Each clique supports the abstraction of a fault-tolerant process group. Dependable applications can then be implemented using the resulting overlay of process groups. CellFarm has a completely decentralized operation, is scalable, and highly flexible, allowing the developer to define the size of each process group. Due to its unique topology, CellFarm offers a natural subtract for supporting efficient replication and to perform load distribution. As a case study, we show how CellFarm can be used to implement a distributed large-scale resource location system where the cliques generated and maintained by CellFarm are used to perform one-hop replication of resource indexes. We show that the structure of CellFarm allows resource location algorithms to perform more efficiently and achieve better load distribution, when compared with a current state-of-art solution.